Dr Adam Gall
A paper delivered at the Association for Academic Language and Learning Conference, November 2023
Introduction
In James Cameron's 1991 blockbuster, Terminator 2: Judgment Day, a sophisticated and intelligent machine is sent from the future by SkyNet, an artificial general superintelligence system, to kill a teenager who after the nuclear apocalypse will become the leader of human resistance to global domination by machines. To help this teenager, John Connor, as well as his institutionalised militant mother, Sarah, to survive this threat, the human resistance of the future sends an older model cyborg (the T-800, played by Arnold Schwarzenegger, first introduced as the monster in Cameron's 1984 film, The Terminator) to defeat the killing machine. I hope I'm not spoiling anything to say that the good guys win, but not in a way that forecloses on sequels, spin-offs, and other storytelling in the fictional universe of the film.
I have no doubt that many of us in the Academic Language and Learning (ALL) space old enough to remember this film have had reason to consider its metaphorical value for our practice and profession recently. We may find ourselves wondering if the Large Language Model (LLM)-based generative tools that our institutions, professional associations and colleagues have been excitedly, or anxiously, engaging with aren't machines from the future sent to kill us or at least to extinguish our livelihoods.
It's true that generative AI language models are also likely doing harm as I speak; this is mostly a much less spectacular technical amplification and partial transformation of social and cultural phenomena that humans have managed to achieve without these tools. According to digital ethicist Per Axbom, these harms include obscured data theft, bias and injustice, carbon cost, concentration of social power, obscured decision-making, accountability projection , misinformation, privacy breaches, and moderator trauma.
Within the context given by that list, to ponder the possibility that LLMs are machines sent from the future to snuff out our livelihoods is not all that far-fetched. It could even be seen as one of the milder potential effects of the widespread adoption of these tools.
We should be careful here, though. Using a cinematic narrative like Terminator 2 to think critically about the effects of AI tools on our work, and on our students' learning, is also potentially harmful. As Axbom also notes, fearmongering sentience claims and imagined harm are also risky. Utopian or dystopian pronouncements about technology that cloud our judgement or allow easy dismissal of meaningful critique should be avoided. Thus, caution is due when using my spectacular and explosive cinematic analogy.
So why, then, have I framed this talk with a discussion of Terminator 2? Firstly, as a sequel, it offers a reconfiguration of the actors and relationships the networks posited in the original. LLMs are also a kind of sequel to the digital humanities and social sciences which, in our field, includes corpus tools and data-driven learning (DDL). What new configurations might this sequel prompt for the actors from earlier periods of ALL digitisation?
Also, the film reminds us that the seeds of the future are sown in the present. It is the present-tense fashioning of socio-technical relationships that matters in the film: the monster Terminator melds with law enforcement and imitates family members; the good guys rely on their friends' loyalty and their own ingenuity and win over ethical scientists. Social institutions and relationships are activated and reactivated but also interrogated throughout the story.
Further, engagement with tech both 'low' and 'high'--isn't limited to the bad guys; the good guys also use technology extensively, including the old-fashioned, 'clunky', Terminator cyborg sent to protect them. This older version of the future may still have some merit. Similarly, I wonder if corpus tools and DDL may not, in their very 'clunkiness', have a persistent attraction for us and even for students.
I will investigate this possibility by considering generative AI-driven learning alongside data-driven learning (DDL) using corpus tools in academic language and writing support. Carrying the film's analogy forward, I ask: could there be a role for the 'clunkier' first version of the digital humanities in a future shaped by smooth efficiency in AI-driven compositional learning?
Corpus tools and Data-driven learning
For those who don't know, corpus tools are software that allow the digital assemblage of textual corpora. That is, they facilitate the collation, processing and coding or marking up of text into data: thousands or millions of words are copied into digital collections and handled in specific ways so they can be used for research or learning. These tools first appeared in the 1960s but became more widespread as desktop computing became economically and socially viable during the 1990s and universities digitised academic work.
Contemporary corpus software usually includes a range of analytical tools as well: concordance searches, collocation searches, n-grams, 'keyword' extraction, as well as parallel corpus functions. In the examples I'm sharing with you [from the subscription version of Sketch Engine tool], we have 'word sketches', which are multidimensional summaries of words (or lemmas) within the corpus chosen. Data-driven learning (DDL) is the use of these tools in classrooms or by students to support their own language learning, especially written academic English.
AI large language models are also based on digital-textual corpora, rather than language as such. That is, a corpus is already an abstraction from language; generative models are trained on concrete, but immense, digital corpora assembled from various sources, including digitised books, social media and the web. This is also an important source of problems such as bias, and a reason why copyright issues are unresolved around LLMs: they have been trained on materials that contain all of the biased or unjust representations the human producers of the corpora have brought to them; also, they have clearly been trained on copyrighted materials.
To return to the Terminator 2 analogy: if the LLMs are the T-1000s, the efficiently liquid and smoothly functional, then our corpus tools could be understood as the T-800s of the current moment: they are older, still potentially powerful tools; their efficacy is well-documented; they are also 'clunky', prone to malfunction, and preserve a vulnerable layer of living human-like tissue that can be injured.
Okay, so that last point might be taking the analogy too far. But human intervention and decision-making about assembling and analysing corpora using these tools is visible and replicable. It is not hidden in a proprietary 'black box'. Further, their socio-technical nature is impossible to smoothly efface in everyday use: much like Arnold Schwarzenegger they don't pass very effectively as discrete actors in their own right (LLMs do, especially in their guise as chatbots and digital assistants).
What do they have in common?
|
Corpus-tool driven DDL and Generative AI driven teaching and learning |
|
Developed for non-pedagogical reasons |
|
'Marketed' for non-pedagogical reasons |
|
Not only technical tools, but also socio-technical systems of relationships |
|
'Qualitative' products out of quantitative processes |
|
Epistemology rooted in empiricism and probability |
|
Relies on textual corpora (large to enormous datasets) |
|
Students can access independently |
How do they differ?
|
Corpus-tool driven DDL |
Generative AI driven teaching and learning |
|
Researcher-led |
Commercially led |
|
User experience (UX) does not lead design or integration |
UX-forward tool design and integration |
|
Established uses motivated by pedagogical values as well as curiosity |
Established uses motivated by values of efficiency/productivity as well as curiosity; emergent uses motivated by pedagogical values |
|
Established evidence base for pedagogical use |
Emergent practical guidance for pedagogical use; questionable (thus far) evidence base for pedagogical use |
|
Slow arrival, uneven interest from higher education institutions |
Rapid, enthusiastic (/anxious) uptake by higher education institutions |
|
'Frozen' corpora or 'living' corpora (in potentia): platform-dependent |
'Frozen' corpora but with commercialised data priming and other services |
|
Technical and algorithmic transparency |
Technical and algorithmic 'black-boxing' |
|
Small, discrete programs; large datasets |
Cloud computing and 'big data' |
|
Open source/non-commercial/semi-commercial (subscription) mixture (research 'customers') |
Commercial tools, mostly subscription model (corporate and professional customers) |
Notes on this comparison and contrast
Some key things in common, then. The first is that neither data-driven learning nor generative AI come out of efforts to support writing instruction or student language learning, at least initially. Corpus-tool driven DDL begins among researchers, as part of a general (and uneven) digitisation of the humanities and social sciences. Secondary applications came via the research-teaching nexus, as the applied linguists who also worked in ALL spaces saw connections between their own research practice and student learning. DDL promised concrete evidence in favour of linguistic decisions that students might make, and also the student-as-language researcher might then be able to work autonomously once relevant skills and resources were in place. This was, they reasoned, a good way of negotiating between the descriptive or empirical and the normative, as well as between autonomous student inquiry and guided student productivity. In Australian universities, their use has been relatively common, but far from ubiquitous (Bednarek et al. 2020).
Another important commonality is that both DDL and LLM-facilitated learning are part of a socio-technical system of relationships. In the case of DDL, the relationships that sustain the tools can be mapped more easily: they depend on the same relationships that sustain the university as a socio-technical institution and that have held, to a greater or lesser extent, for a quarter century or more. Critiques of the corporatised, digitised university continue to apply; the material undergirding of everyday networked computing in resource extraction and human exploitation is still at stake here. At the same time, the digital humanities has fostered forms of accessibility to data that are unprecedented in the history of universities. We can see the products of this in the digitisation of documents and archives, gallery and museum collections, which has even influenced undergraduate teaching. The corpora activated in DDL work similarly: they are transformed into data and become a sort of quasi-commons, mediated by the corpus tools, which tend to be open or semi-open access.
By contrast, making visible how AI is part of a system of socio-technical relationships, rather than a discrete technical entity, has been a difficult achievement of critical research over recent years. A good example of this is Kate Crawford's Atlas of AI, which argues persuasively that artificial intelligence, including LLMs, is both embodied and material and depends entirely on a much wider set of political and social structures ; it is neither (wholly) artificial nor intelligent, but better understood as a registry of power (2021, p.8). That this effort of mapping and evaluation is necessary points to one of the major challenges of engaging with AI not only as researchers, but also as learners or educators: the technology is opaque to non-experts and the tools themselves are (increasingly) proprietary.
The evidence base for corpus-tool driven DDL is by now well established. As Muneera Muftah (2023) reports, corpora are valued by language teachers because they are built from authentic examples of language in use, and they are perceived as efficacious for promoting lexico-grammatical awareness through curating examples and bringing patterns into focus. A meta-analysis of dozens of empirical studies by Boulton & Cobb (2017) showed consistent positive effects on second language learning especially in learning to write. A later meta-analysis by Lee and colleagues (2019) showed positive effects of corpus tool use on vocabulary knowledge, as well as referential meaning and lexico-grammatical confidence. These effects are particularly strong with written language.
One common technique is provision of concordance lines either teacher (indirect DDL) or student (direct DDL) generated allowing students' guided discovery of lexico-grammatical patterns. Students with more time and curiosity can also explore both free, online tools for lexico-grammatical insights (SKELL) or can quantify features of their own manuscripts (COCA). The corpus as a reference point for interpreting such quantification means that students are not limited to use of abstract readability measures (e.g. Flesch-Kincaid) but are able to apprehend the similarities and differences between their own written style and 'typical' academic styles in fuller complexity.
Less research has been done on L1 learners and corpus tools, but it has been noted that prior language proficiency influences the degree of effectiveness of tool use for language learning; L1 learners can also deploy these tools for lexico-grammatical insights as they move into the discourse communities of their disciplines.
Although we have come to see ChatGPT as a transformative moment, digital language learning and writing scholars have built up a prior critical scholarship about AI. I won't go into this in detail, but an example would be Boyle's (2016) argument for writing as posthuman practice which anticipates the ubiquity of LLMs. More recently, educational researchers have become prominent among those investigating AI use within higher education settings (Crompton & Burke, 2023) Further, in the period between 2016 and 2022, a little less than a fifth of research has been on AI use in language learning, broadly defined; over the same period, research has overwhelmingly focused on student use of AI.
Evidence for the value of LLM tools to student learning has also been produced, though empirical studies tend to emphasise the likely efficacy of tools for specific writing or language tasks, rather than vindicating their role as facilitators of student learning. Kohnke et al. (2023) examine affordances of ChatGPT for language teaching and learning and the competencies its use is likely to require is more typical of the literature so far. The authors point out the ways that LLM outputs align with desirable language acquisition inputs: for example, ChatGPT can generate rich texts, conversational practice, immediate feedback and simulated authenticity. Empirical studies examining LLM outputs in relation to specific language and writing functions, such as Fang et al. 2023 on grammatical error correction, tell us very little about how well these tools might support student learning of grammar for meaning, or command over academic style or lexical choices. An evidence base for student learning in areas of writing or other academic skills is not yet extant.
A further potential for AI tools is their engagement of students in autonomous learning. I can certainly see this in AI-driven tools such as Grammarly: both higher education researcher, Inger Mewburn, and my ALL colleague, Lan Nguyen have made the case. With respect to generative tools, student autonomy may sometimes be at odds with student learning, especially in relation to writing and composition. Although the language of 'prompt engineering' implies an empowered position for users as 'prosumers' of the technology, the set of relationships being established here are not designed primarily to promote either learning or autonomy, at least not beyond certain parameters. Indeed, generative AI's development as a commercial system of relationships means that any promise of autonomy, may be more closely aligned with other values such as efficiency and productivity and linked to a generalised deskilling of writing (and to some extent reading). DDL, in contrast, positions the student as researcher closer to the conventional usage of the term 'engineer' as one who builds or maintains machines, rather than simply uses them.
Changing economies of text
This brings us to a key point distinguishing the established and emergent sets of socio-technical relationships. Where is value produced and located in each? Which dispositions do these systems produce? A glance at the history of digital writing scholarship may help us here.
A decade and a half ago, literary scholar and teacher, Bruce Gardiner (2007), surveyed digital writing scholarship and found it wanting. The qualitative transformations that researchers claimed for digital composition and for associated technologies such as hypertext, amplified and externalised endemic features of textuality. Although Gardiner did not comment on corpus tools, the implications of his argument extend there as well: text as data is already a feature of our engagement with 'analogue' text; if we want to, we can count words, look for and notice lexico-grammatical patterns. (The low-tech version of this is, perhaps, through the authentic exemplar).
Gardiner pointed out that it was the political economy of text being transformed by digitisation. Some practices would no longer be accorded the same economic or institutional value as quantity replaced quality: issues of speed, efficiency and productivity were what was at stake, not textuality per se (nor, for that matter, the fundamentals of reading practice or learning to write). The impossible necessity of communicating intersubjectively through writing did not thereby evaporate, or even fundamentally change, but aspects of the process would now have a much lower market value.
Within a decade of these debates on digital writing, writing studies scholars posited that one effect of this transformation of political economy is writing as content (Dush, 2015). This conceptualisation was another argument for a shift in textuality and the need for students and teachers to understand language and writing in new terms. For Lisa Dush (2015), this shift was modelled on the content creation and management as key professional and commercial arenas for students to activate writing and editing skills beyond the university. Content was text as conditional, computable, networked and commodified; student's learning is thus reoriented towards socio-technical relationships shaped by a changing political economy of text.
This process of text becoming content, I believe, is amplified by LLMs to the point where, rather than repurposing language learning and especially the learning of writing to find value in adaptability, the products of such learning are themselves being devalued, their production deskilled. LLMs can generate great reams of content on any topic, driving down the market value of text; more importantly they transform our relationship to the time it takes to create meaningful text of our own especially the time it takes to learn to do this well. The values LLMs embody, at least until further notice, are values of productivity, speed and efficiency.
By contrast, the 'clunkier' corpus tools of DDL fit a space of writing as inquiry, both inquiry about disciplinary subject matter, and about language itself. In the past, many of us balked at the rendering of textual information as data. Yet conceiving of text as data for student inquiry still has the potential effect of rendering the writing process as one of finding out, of understanding how and why, but also engaging in a process that is valuable for itself. In the current context, corpus tools may be a site that combines remediation of language as data and the valuation of academic writing as process, not product.
Conclusion
Through this detour into discussions about digital writing and composition, I've begun to make the argument that the set of relationships around 'clunkier' technologies also connect to important shared values. By contrast, LLM-based tools and the socio-technical relationships they engage reinforce other values, such as productivity.
Of course, students also value productivity, and even discourse that is explicit about this as the guiding value of generative AI will be attractive to them. Indeed, as Maggie Charles (2022) has found, many students proficient in corpus use may stop using the technology due to scarcity of time (p. 1). Further, contemporary higher education imposes values of productivity and efficiency, sometimes as a 'hidden curriculum' though this is often a product of pedagogical priorities (sometimes unintentionally), it is also enforced by the need for organisational efficiency, policy-driven changes and the set of circumstances in which students access higher education, and teachers and advisers work within it.
So where does this leave us? Before returning to Terminator 2, I want to point out another of my favourite 1990s cultural products, superstar theorists. In this esteemed cohort, Gayatri Spivak is a special case. Spivak's commitment to teaching is almost as important to her academic persona as the complexity of her most well-known articles. For Spivak, the task of the humanities teacher principally, in her case, through close reading is the uncoercive rearrangement of desire (2004). This might be a way to understand our own roles in response to the dominance of values like productivity in higher education. Beyond arguing in our institutions that process matters to learning, we might try to model, foster and encourage other values, and indeed, other desires for our students.
Plainly, there are few examples of this offered by Terminator 2, though we might not expect them in a sci-fi action blockbuster. But the film does offer a model of an alternative value that may be useful: this is the curiosity of the tinker. Indeed, in a recent article on educator engagement with ChatGPT in JALL, McLeod & Richardson (2023) refer to playful tinkering as an important value. Their observation is consistent with my endorsement in this paper of 'clunky' tech: we can only tinker where there are affordances for doing so, where the socio-technical relationships are open. However, the dominant trends in both hardware and software arenas have been towards black-boxing and the prevention of tinkering.
Finally, a note on the metaphors we live (and learn) by. In her critique of 'tool' and 'collaborator' metaphors for framing ChatGPT for student's critical digital literacy learning, Salena Sampson Anderson (2023) points out the potential value of medical metaphors instead. While 'tool' fails to address the ethical value of transparency, and 'collaborator' misrepresents affordances for LLM accountability, bodily metaphors could, she suggests, be more useful. Anderson suggests that ChatGPT be seen more like a 'bodily product' (pp. 8-9), a combination of human viscera and technical intervention, a bit like processed blood products used in transfusions. What exactly are we injecting into our learning? It might be deadly, it might be lifesaving, it might be a form of cheating (like blood doping).
Anderson's preference for liquefied bodily metaphors could also be consistent with the liquidity of the T-1000 Terminator. In which case, Schwarzenegger's T-800 continues to work as an avatar of the older corpus tools and data driven learning: this is the cyborg we might redeem, even if its reappearance is untimely in the larger narratives of tech utopia and dystopia. Even the poorly rendered sentimentality of Schwarzenegger's closing dialogue in T2 Judgment Day ( I know now why you cry ) could be recuperated when we consider 'clunkiness' as an intermediate value encouraging curiosity and learning. This moment becomes more sympathetic if we identify with the vulnerability and ingenuousness of the language learner; the Terminator, after all, has learned to communicate something new. It uses that learning to communicate authentically with the human characters, but also to remind them of the disanalogy between their learning and its own but it is something I can never do .
Anderson, Salena Sampson. ' Places to Stand : Multiple Metaphors for Framing ChatGPT's Corpus'. Computers and Composition 68 (1 June 2023): 102778. https://doi.org/10.1016/j.compcom.2023.102778.
Per Axbom, 'If a hammer was like AI', June 22, 2023; accessed November 6, 2023, https://axbom.com/hammer-ai/
Per Axbom, 'The Elements of AI Ethics', June 2, 2023; accessed November 6, 2023,
Bednarek, Monika, Peter Crosthwaite, and Alexandra I. Garcia. 'Corpus Linguistics and Education in Australia'. Australian Review of Applied Linguistics 43, no. 2 (3 July 2020): 105-16. https://doi.org/10.1075/aral.00029.edi.
Boulton, Alex, and Tom Cobb. 'Corpus Use in Language Learning: A Meta-Analysis'. Language Learning 67, no. 2 (2017): 348-93. https://doi.org/10.1111/lang.12224.
Boyle, Casey. 'Writing and Rhetoric and/as Posthuman Practice'. College English 78, no. 6 (2016): 532-54.
James Cameron (director), Terminator 2: Judgment Day, 1991.
Charles, Maggie. 'The Gap between Intentions and Reality: Reasons for EAP Writers' Non-Use of Corpora'. Applied Corpus Linguistics 2, no. 3 (1 December 2022): 100032. https://doi.org/10.1016/j.acorp.2022.100032.
Kate Crawford, The Atlas of AI: Power, politics and the planetary costs of artificial intelligence, Yale, 2021.
Crompton, Helen, and Diane Burke. 'Artificial Intelligence in Higher Education: The State of the Field'. International Journal of Educational Technology in Higher Education 20, no. 1 (24 April 2023): 22. https://doi.org/10.1186/s41239-023-00392-8.
Dush, Lisa. 'When Writing Becomes Content'. College Composition and Communication 67, no. 2 (2015): 173-96.
Fang, Tao, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, and Yue Zhang. 'Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation'. arXiv, 4 April 2023. https://doi.org/10.48550/arXiv.2304.01746.
Gardiner, Bruce. 'What's New and How New Is It? Contemporary Rhetoric, the Enlightenment, Hypertext and the Unconscious'. In What Is The New Rhetoric?, by Susan Thomas, 29-45. Newcastle-upon-Tyne, UNITED KINGDOM: Cambridge Scholars Publishing, 2007.
Kohnke, Lucas, Benjamin Luke Moorhouse, and Di Zou. 'ChatGPT for Language Teaching and Learning'. RELC Journal 54, no. 2 (1 August 2023): 537-50. https://doi.org/10.1177/00336882231162868.
Lee, Hansol, Mark Warschauer, and Jang Ho Lee. 'The Effects of Corpus Use on Second Language Vocabulary Learning: A Multilevel Meta-Analysis'. Applied Linguistics 40, no. 5 (1 October 2019): 721-53. https://doi.org/10.1093/applin/amy012.
McLeod, Amber, and Hannah Richardson. 'Co-Constructing Skills for ChatGPT at University'. Journal of Academic Language and Learning 17, no. 1 (11 September 2023): T70-80.
Muftah, Muneera. 'Data-Driven Learning (DDL) Activities: Do They Truly Promote EFL Students' Writing Skills Development?' Education and Information Technologies 28, no. 10 (2023): 13179-205. https://doi.org/10.1007/s10639-023-11620-z.
Gayatri Chakravorty Spivak, 'Terror: A Speech After 9-11', boundary 2, Vol 31, No 2, Summer 2004, 81-111.
Wilson, Owen, and Joshua Billam. 'ChatGPT in June 2023: An Annotated Bibliography for ALL Professionals'. Journal of Academic Language and Learning 17, no. 1 (14 June 2023): T9-15.